Qualitative Assessment in the Chemical Engineering Curriculum

Background Assessment is an important part of student learning. Probably the most important part since the method of assessment has a major influence on the way students accomplish their studies. It is very important to use this fact in order to create the best possible circumstances for student learning. If we want learning to be more qualitative than quantitative, deep oriented and not surface oriented, to focus on the curriculum as a whole we educate professional chemical engineers assessment must always be designed with this in mind. Students must be given the opportunities to demonstrate quality, a deep orientation, and comprehensive views on these occasions. In recent years we have carried out major changes within the chemical engineering curriculum (Bachelor of Science level) at Lund University. The most important changes comprise a comprehensive view of the curriculum, including a deep orientation of teaching and learning, fewer and more comprehensive courses, and a carefully prepared schedule of courses more focused on food and pharmaceutical technology. Furthermore, "non-technical" elements such as written and verbal communication skills, engineering ethics, quality assurance, economics, environmental problems and social psychology have been introduced into our curriculum. These important items are introduced in an introduction course during the first year and are then integrated in different courses throughout the curriculum. Finally, we have introduced carefully prepared and formulated educational objectives knowledge, skill and attitude at all levels within the curriculum. An important and serious problem is that the assessment has not changed very much and in many cases does not correspond to our educational objectives. To put it simply, our students are not assessed against the comprehensive view of the curriculum expressed in the educational objectives. There is an apparent risk that student learning is surface oriented and only aimed at reproducing facts. In a recently completed project at LTH School of Engineering in Helsingborg, Lund University, current examination forms have been investigated and a test with qualitative examination has been performed (Olsson, 2000). The results of the project partly confirm the apprehensions about the assessment. Pedagogical problem The pedagogical problem is actually very simple and yet so difficult. How do we design and perform an assessment so that it creates the best possible circumstances for student learning? Naturally, the optimal assessment is a combination of different examination forms with variations between the different parts of a course and between courses within the curriculum. But in general, the assessment should require the students to demonstrate whether the qualitative goals have been achieved. General project idea The general project idea is that we must change examination so that it becomes more qualitative than quantitative. An examination must focus on "how well" a subject is mastered rather than "how much" of a subject that has been acquired. The examination should stimulate a deep oriented, holistic learning that focuses on the overall goals of the curriculum. Our assessment must be more oriented towards the engineering profession. We educate chemical engineers. This could be regarded as a kind of authorization but it is not included in the examinations. It would be very interesting to perform tests with external examiners with this aspect in mind. Assessment of attitudes should also be included. Professional engineers are of course suitable but why not also use university teachers from other faculties such as medicine and social sciences? In different medical disciplines the attitudes of the students are crucial and experiences from these areas could certainly be used in a modified form for our purposes. We must assess practical engineering skills. This includes laboratory skills and planning of experimental work. Some kind of proficiency tests could be used for certain practical parts of the educational programme. After passing such an examination the student could receive a licence for the practical skills of that part of the chemical engineering education. Assessment of practical laboratory skills also has many connections with other areas of the university. The Art Academy, the Theatre Academy and the Faculty of Arts and Theology most certainly use many methods and have experiences that could generate new useful ideas for an engineering education. How is a work of art or a poem assessed? Many untraditional contacts will be taken during this project. We must test a combination of formative and summative examination forms. This is especially important within the Faculty of Engineering. For reasons of history and administration, above all extensive teaching of courses in parallel, a system has evolved where all examinations are concentrated to special examination weeks four times a year. There are many convincing pedagogical arguments for altering this rigid system. Any reform in this direction necessitates the introduction of new assessment methods. This project will facilitate changes in the educational system at the Faculty of Engineering. In several courses the students work in project groups to solve different assignments. How are individual students assessed when they are part of a group? This teaching method is used throughout most courses at Aalborg University in Denmark and experiences from there could be used in this project. We must introduce a more comprehensive assessment with examinations that cover several courses. This is especially important towards the end of the education. The educational programme and especially the engineering profession is an entirety. To sum up we should use a variety of different assessment methods. There is no overall way of assessing that will solve all examination problems. However, taken together the proposed different methods will give us a better assessment than we have today. We expect them to be an improvement because they will better correspond to the educational objectives of the curriculum and, thereby, increase student learning, which is the principal purpose of all pedagogical activities at a university. Aim of the project The aim of the project is to develop, test and evaluate various forms of qualitative assessment methods. Special aspects that will be considered are the influence of assessment methods on students with different ethnical background and on older students with work experience. One fifth of the students that were accepted for the chemical engineering education (1999) do not have Swedish as their native language. In this project we will especially investigate the effects of different examination forms on non-Scandinavian students. Less than 20% of the students have entered the university immediately after completing upper secondary school and about 65% of the students have worked for shorter or longer periods before they started their chemical engineering studies at the university. This means that we have many students that are older than corresponding students at the Master of Science level. Many students also have longer work experiences when they enter the university. These facts will also be given special attention. A very important objective of the project is an international distribution of the results. This will be guaranteed through publication of all results in pedagogical papers and/or presentations at various conferences. Theoretical basis Within this project the SOLO taxonomy will be used to make qualitative judgments during planning and evaluation of different examination methods. The SOLO (Structure of the Observed Learning Outcome) taxonomy is a model for qualitative evaluation of teaching and examination (Biggs and Collis, 1982). It consists of different levels of increasing structural complexity. Students intellectual development through the curriculum will be investigated using Perry ́s Scheme of Intellectual and Ethical Development (Perry, 1970). This scheme consists of different stages characterized in terms of students' attitudes towards knowledge. Methods and time planning Different assessment methods will be developed and tested mainly on students from the chemical engineering education (Bachelor of Science level) at Lund University. The length of this programme is three years with a total of 100-120 students. Some of the tests could be extended to the chemical engineering education at Master of Science level and perhaps also to other engineering programmes. During the first and second year of the project preliminary studies will be performed and different examination ideas will be developed into practically useful examination tasks. Different tests in smaller scales will be performed. Continuous documentation is an important part of the work and some reports and presentations should be ready during the first year. During the third year the work will be concentrated towards comprehensive tests of various assessment methods. All students and several of the teachers representing different subjects within the curriculum will be involved in these tests. Evaluation, documentation and the presentations of the results are important parts of the work. Project participants The Project director is Dr Thomas Olsson, LTH School of Engineering in Helsingborg, Lund University. Thomas Olsson has participated very actively in undergraduate education at LTH, both as examiner for several chemical engineering courses, and as the director of undergraduate studies in chemical engineering at the Bachelor of Science level. A reference group (project group) has been formed. Aside from the project director, the other participants are Dr Jan Hellberg, Centre for Teaching and Learning at Lund University and Department of Occupational Therapy, Professor Peter Arvidsson, Centre for Teaching and Learning at Lund University and Department of Media and Communication Studies, Professor Anders Axelsson, Department of Chemical Engineering and Jonas Kronkvist, 3rd year student at the chemical engineering curriculum (Bachelor of Science level) at Lund University. Documentation and evaluation The main outcome of the project will be the actual examinations developed for the purposes described above. The results will be documented in reports and articles in pedagogical and/or engineering periodicals and presented at national and international conferences. The project will be continuously evaluated through questionnaires and interviews with students and teachers within the chemical engineering curriculum. Project activities during the academic year 2000/2001 The project was presented in a lecture at a conference for chemical education, SPUCK XI (Sveriges Pedagogiska Universitetskemisters Centrala Konferens), held at Lund University, Centre for Chemistry and Chemical Engineering, 16th-18th August 2000. A poster, "Qualitative Assessment in the Chemical Engineering Curriculum", was presented at the 8:th International Improving Student Learning Symposium, Improving Student Learning Stratigically, held at UMIST, Manchester, 4th-6th September 2000. A paper, "Assessment of Experimental Skills and Creativity Using a Modified OSCE-method a Summative Performance-Based Examination in Chemical Engineering", was presented at the 9:th International Improving Student Learning Symposium, Improving Student Learning using Learning Technologies, held at Heriot-Watt University, Edinburgh, Scotland, 9th-11th September 2001. Methods for assessments of skills and creativity have been developed (Olsson, 2002): Most courses in a chemical engineering curriculum include practical experimental parts. These parts are normally assessed formatively in the laboratory. Students hand in reports and demonstrate their assignments and they get immediate feedback. This is very important and commendable. However, summative assessments of practical engineering skills are of rare occurrence in engineering curricula. An individual summative assessment could be of major importance to influence students to focus on the skill objective of the curriculum. Medical education all over the world uses a summative performance-based examination called "Objective Structured Clinical Examination, OSCE (Harden et al., 1975). The aim of the OSCE is to test students' clinical and communication skills in a planned and structured way. The examination consists of several stations each presenting a scenario. At each station an examiner is observing the student's performance. The result is decided by judging how well the performance meets a number of stated criteria. Can these ideas of assessment be used in a chemical engineering curriculum? The OSCE-method takes considerable resources. The paper presents a study of an assessment of experimental skills and creativity in chemical engineering using a modified OSCE-method. A typical examination will last for 3-4 hours and consists of 6-8 different stations. More than 25 different tasks have so far been constructed. They test students' experimental skills, planning of experimental work, critical and reflective thinking and creativity and they are constructed so that they will require students to combine knowledge and skill to perform a task. It is important that most of the tasks are open-ended to allow students to show different qualitative approaches (Biggs and Collis, 1982). Students will be asked to discuss and explain ideas and procedures formulate and test hypotheses, design experiments etc. students must perform their understanding. The results of the examination tests are investigated using both qualitative and quantitative approaches. The qualitative part comprises the use of different focus groups, with students participating in the summative performance-based examination and a reference group. The quantitative studies are performed using a specially designed questionnaire investigating attitudes, intellectual development (Perry, 1970) and approaches to learning. Some features of the method are: · a summative performance-based assessment increases the students' awareness of the over-all objectives of the curriculum · a performance-based assessment allows students to demonstrate a rich array of abilities · a performance-based assessment allows the examiner to get a more complete picture of a student's abilities and it facilitates effective feedback on student performance · there is a positive correlation between summative performance-based assessment and students' deep approaches to learning especially the occurrence of tasks requiring creativity and planning of experimental work favours a deep approach. Project activities during the academic year 2001/2002 Assessment methods that foster integration of non-technical skills in a chemical engineering curriculum through experiential learning (Kolb, 1984) have been developed: The design of the curriculum includes an accurately prepared schedule of integrated courses supporting a deep orientation of teaching and learning and an integration of non-technical skills and competencies such as communication skills, engineering ethics, quality assurance, applied economics, environmental issues and social psychology. These important items are introduced in an introduction course during the first year and are then integrated throughout the curriculum. Formative performance assessments include rhetorical speeches, case studies, scientific papers, poster presentations, standard operating procedures (SOP), ethical investigations and field observations. Besides written reports all activities are presented orally at seminars with formal opposition from teachers and other students. A paper, "A Combined Formative Performance Assessment and Summative Reflective Assessment Fostering Experiential Learning and Integration in an Engineering Curriculum", will be presented at the International Research Conference: Learning Communities and Assessment Cultures: Connecting Research with Practice jointly organised by the EARLI Special Interest Group on Assessment and Evaluation and the University of Northumbria, 28th-30th August 2002, University of Northumbria. The work presented in this paper investigates how a reflective assessment influences the experiential learning promoted by the performance-based assessments and how this affects students' integrative abilities. The purpose of reflection is to learn from experiences. Students write reflective papers that are personal, self-reflective and focus on knowledge, skills and attitudes acquired during the introduction course. They reflect on how they will use these competencies in engineering courses and in future professional life. They also reflect on learning and how to improve as learners. The combination of formative and summative assessment methods favours experiential learning as described by Kolb's learning cycle. Formative performance assessments give several concrete experiences that are reflected upon and conceptualised in the summative reflective assessment. An active experimentation occurs when new competencies are integrated and applied in engineering courses that in turn results in new experiences. Some features of the method are: · a formative performance assessment of non-technical skills and attitudes allows students to demonstrate different abilities and facilitates feedback on student performance · the combination of a formative performance and summative reflective assessment increases the quality of the learning process · many students possess latent integrative abilities and integration of nontechnical skills in different engineering courses is favoured by a reflective assessment · metacognition is favoured by the use of a reflective assessment and metacognitive skills influence the student learning process throughout the curriculum · the proposed assessment procedure is more oriented towards quality assurance of learning outcomes than just testing of knowledge and skills (quality control) Planned activities for the academic year 2002/2003 The main ideas for the final year of the project include more comprehensive assessments with examinations that cover several courses. Assessment methods more oriented towards the engineering profession and assessments of attitudes and intellectual and ethical development (Perry, 1970) will also be developed. References Biggs, J. B. and Collis, K. F. (1982). Evaluating the Quality of Learning. The SOLO Taxonomy (Structure of the Observed Learning Outcome), Academic Press, New York Harden, R. M., Stevenson, M., Wilson Downie, W. and Wilson, G. M. (1975). Assessment of Clinical Competence Using Objective Structured Examination, British Journal of Medical Education, 1, 447-451 Kolb, D. A. (1984). Experiential Learning: Experience as the Source of Learning and Development, Englewood Cliffs, New Jersey: Prentice-Hall Olsson, T. (2000). Qualitative Aspects of Teaching and Assessing in the Chemical Engineering Curriculum Applications of the SOLO Taxonomy, Paper presented at the 7th International Improving Student Learning Symposium, University of York, 1999, In Rust, C. (Ed.), pp. 304-324, Improving Student Learning Through the Disciplines, The Oxford Centre for Staff and Learning Development Olsson, T. (2002). Assessment of Experimental Skills and Creativity Using a Modified OSCE-method a Summative Performance-Based Examination in Chemical Engineering, Paper presented at the 9th International Improving Student Learning Symposium, Heriot-Watt University, Edinburgh, Scotland, 2001, In Rust, C. (Ed.), pp. 310-323, Improving Student Learning Using Learning Technologies, The Oxford Centre for Staff and Learning Development Perry, W. (1970). Forms of Intellectual and Ethical Development in the College Years: A Scheme, Holt, Rinehart and Winston, New York Qualitative Assessment in Engineering Education